1. Field of the Invention
The present invention relates to an imaging apparatus, such as a video camera or a digital still camera, in which an XY address scanning-type imaging device (imager, image sensor) such as a CMOS (Complementary Metal Oxide Semiconductor) imaging device is used, and to a method for reducing fluorescent lamp flicker which occurs in a video signal obtained from the imaging device when a subject is photographed by the imaging apparatus under the illumination of a fluorescent lamp.
2. Description of the Related Art
When a subject is photographed by a video camera under the illumination of a fluorescent lamp which is powered on by a commercial AC power-supply, an intensity change with respect to time, that is, so-called fluorescent lamp flicker, occurs in the video signal of the photographed output due to the difference between the frequency (twice the commercial AC power-supply frequency) of the luminance change (change in the amount of light) of the fluorescent lamp and the vertical synchronization frequency (imaging frequency) of the camera.
A description will now be given of a case in which, for example, in an area where the commercial AC power-supply frequency is 50 Hz, a subject is photographed by an NTSC CCD camera (the vertical synchronization frequency (the field frequency in this case) is 60 Hz) under the illumination of a fluorescent lamp of a non-inverter system (although it is not restricted to the case of a non-inverter-system fluorescent lamp because flicker occurs also in the case of an inverter-system fluorescent lamp when rectification is not sufficient). In this case, as shown in FIG. 1, whereas the period of one field is 1/60 seconds, the period of the luminance change of the fluorescent lamp is 1/100 seconds. As a result, the exposure timing of each field is shifted with respect to the luminance change of the fluorescent lamp, and the amount of exposure in each pixel changes.
For this reason, for example, when the exposure time is 1/60 seconds, in periods a1, a2, and a3, the amount of exposure differs even with the same exposure time. Furthermore, when the exposure time is shorter than 1/60 seconds (when it is not 1/100 seconds, as will be described later), in periods b1, b2, and b3, the amount of exposure differs even with the same exposure time.
The exposure timing with respect to the luminance change of the fluorescent lamp returns to the original timing every three fields, and therefore, the intensity change due to flicker repeats every three fields. That is, the luminance ratio of each field changes depending on the exposure period, but the flicker period does not change.
However, in a progressive-system-type camera, such as a digital still camera, when the vertical synchronization frequency (the frame frequency in this case) is 30 Hz, the intensity change is repeated every three frames.
Furthermore, for the fluorescent lamp, usually, a plurality of fluorescent substances, for example, red, green, and blue fluorescent substances, are used to emit white light. However, each of these fluorescent substances has specific persistence characteristics and emits light in a decaying manner with individual persistence characteristics in the period from the stopping of discharge, which exists in the period of the luminance change, up to the start of the next discharge. For this reason, in this period, since the light, which is initially white light, gradually decays while changing its hue, if the exposure timing is shifted in the manner described above, not only does the intensity change but a hue change also occurs. Furthermore, since the fluorescent lamp has specific spectral characteristics such that a strong peak exists at a specific wavelength, variation components of the signal differs depending on the color.
Then, so-called color flicker occurs due to such a hue change and the difference in the variation components for each color.
In comparison, when the power-supply frequency is 50 Hz and the vertical synchronization frequency of the imaging apparatus is 60 Hz, as shown in FIG. 1, if the exposure time is set to 1/100 seconds, which is the period of the luminance change of the fluorescent lamp, as shown in the bottommost portion of FIG. 1, the amount of exposure becomes constant regardless of the exposure timing, and flicker does not occur.
Furthermore, a method for reducing fluorescent lamp flicker without specifying the shutter speed in this manner has been considered. In the case of an imaging apparatus, like a CCD imaging apparatus, in which all the pixels within one picture plane are exposed at the same exposure timing, since an intensity change and a color change due to flicker occur only between fields, a reduction in flicker can be realized comparatively easily.
For example, in the case of FIG. 1, if the exposure time is not 1/100 seconds, flicker occurs at the repetition period of three fields. Therefore, flicker can be suppressed to a level at which there is no problem in practical terms by predicting the current luminance and color changes from the video signal three fields before so that the average value of the video signals of each field becomes constant and by adjusting the gain of the video signal of each field according to the predicted result.
However, in an XY address scanning-type imaging device, such as a CMOS imaging device, the exposure timing for each pixel is shifted in sequence by an amount corresponding to one period of the reading clock (pixel clock) in the horizontal direction of the picture plane, and the exposure timing differs in all the pixels. As a result, in the above-described method, flicker cannot be suppressed sufficiently.
FIG. 2 shows this situation. As described above, also in the horizontal direction of the picture plane, the exposure timing of each pixel is shifted in sequence, but one horizontal period is sufficiently short when compared to the period of the luminance change of the fluorescent lamp. Thus, assuming that the exposure timings of the pixels in the same line are the same time, the exposure timing of each line in the vertical direction of the picture plane is shown. In practical terms, the above assumption does not cause problems to occur.
As shown in FIG. 2, in an XY address scanning-type imaging apparatus, for example, a CMOS imaging apparatus, the exposure timing differs for each line (F0 indicates the situation for a particular field), and the amount of exposure differs in each line. As a result, an intensity change and a color change due to flicker occur not only between fields but also inside fields, and the changes appear as a stripe pattern (the direction of the stripes themselves is the horizontal direction, and the direction of the change of the stripe is the vertical direction) on the picture plane.
FIG. 3 shows the state of this in-plane (intra-picture plane) flicker when the subject is a uniform pattern. Since one period (one wavelength) of the stripe pattern is 1/100 seconds, stripe patterns for 1.666 periods occur in one picture plane. When the number of reading lines per field is denoted as M, one period of the stripe pattern corresponds to L=M* 60/100 at the number of reading lines. In the specification and the drawings, an asterisk (*) is used as a symbol for multiplication.
As shown in FIG. 4, this stripe pattern corresponds to five periods (five wavelengths) in three fields (three frames), and when viewed continuously, it appears to flow in the vertical direction.
FIGS. 3 and 4 show only the intensity change due to flicker. However, in practice, the above-described color change is added, and the image quality is deteriorated considerably. In particular, color flicker becomes more noticeable as the shutter speed becomes higher (the exposure time becomes shorter), and in the XY address scanning-type imaging apparatus, the influence of the color flicker appears within the picture plane. Consequently, the deterioration of the image quality becomes more pronounced.
Also, in the case of such an XY address scanning-type imaging apparatus, when the power-supply frequency is 50 Hz and the vertical synchronization frequency of the imaging apparatus is 60 Hz, as shown in FIG. 2, if the exposure time is set to 1/100 seconds, which is the period of the luminance change of the fluorescent lamp, the amount of exposure becomes constant regardless of the exposure timing, and fluorescent lamp flicker, including in-plane flicker, does not occur.
Furthermore, a method for reducing fluorescent lamp flicker that is specific to the XY address scanning-type imaging apparatus, such as a CMOS imaging apparatus, without specifying the shutter speed in this manner, has been proposed.
More specifically, in Japanese Unexamined Patent Application Publication No. 2000-350102 or 2000-23040, a method is disclosed in which flicker components are estimated by measuring the amount of light from the fluorescent lamp by using a photoreceiving device and a photometering device, and the gain of a video signal from the imaging device is controlled in accordance with the estimated result.
However, when a subject is photographed under the illumination of a fluorescent lamp by means of an XY address scanning-type imaging apparatus such as a CMOS imaging apparatus, the form of flicker which occurs in the video signal from the imaging apparatus is greatly changed according to the combination of the video system of the imaging apparatus (specifically, the vertical synchronization frequency), the frequency of the commercial AC power-supply for driving the fluorescent lamp, and the shutter speed (exposure time) of the electronic shutter.
More specifically, as the video system of the imaging apparatus, the NTSC system (the vertical synchronization frequency is 60 Hz) and the PAL system (the vertical synchronization frequency is 50 Hz), which correspond to a broadcasting system, are known. Most recent video cameras are compatible with both the NTSC system and the PAL system. It is common practice that, when video cameras are shipped from the factory, they are electrically set to either the NTSC system or the PAL system according to the shipment destination.
The commercial AC power-supply frequency is 50 Hz in some areas of Japan and in some countries or areas of the world, and it is 60 Hz in other areas of Japan and in other countries or areas of the world.
Case 1 of FIG. 5A shows a case in which a subject is photographed by a CMOS imaging apparatus of the NTSC system under the illumination of a fluorescent lamp in an area where the power-supply frequency is 50 Hz.
In this case, whereas one field is 1/60 seconds, the period of the luminance change of the fluorescent lamp is 1/100 seconds. Consequently, as shown in FIGS. 2 to 4, during the normal shutter time, where the exposure time is 1/60 seconds, and also during the high-speed shutter time, where the exposure time is shorter than 1/60 seconds, flicker having continuity in the time axis, that is, flicker whose repetition period is three fields (three frames), occurs (when viewed continuously, it appears to flow in the vertical direction).
However, as shown in case 1 in FIG. 7, when the shutter is set to a high-speed shutter with an exposure time of 1/100 seconds, the amount of exposure becomes constant regardless of the exposure timing, and flicker, including in-plane flicker, does not occur.
Case 2 of FIG. 5B shows a case in which a subject is photographed by a CMOS imaging apparatus of the PAL system under the illumination of a fluorescent lamp in an area where the power-supply frequency is 60 Hz.
In this case, whereas one field is 1/50 seconds, the period of the luminance change of the fluorescent lamp is 1/120 seconds. Consequently, during the normal shutter time, where the exposure time is 1/50 seconds, and also during the high-speed shutter time, where the exposure time is shorter than 1/50 seconds, flicker having continuity in the time axis, that is, flicker whose repetition period is five fields (five frames), occurs (when viewed continuously, it appears to flow in the vertical direction).
However, as shown in case 2 in FIG. 7, when the shutter is set to a high-speed shutter with an exposure time of 1/120 seconds or 1/60 seconds, the amount of exposure becomes constant regardless of the exposure timing, and flicker, including in-plane flicker, does not occur.
As in case 1 or case 2 of FIG. 7, when flicker having continuity in the time axis, that is, flicker whose repetition period is a plurality of vertical periods (a plurality of picture planes), occurs, flicker components can be reduced by the above-described conventional method or the method of the invention-of the earlier application (Japanese Patent Application No. 2003-173642) by the same inventors as those of the present application (to be described later), in which flicker components are estimated using the continuity of flicker, and the video signal from the imaging device is corrected in accordance with the estimated result, thereby reducing the flicker components.
In comparison, case 3 of FIG. 6A shows a case in which a subject is photographed by a CMOS imaging apparatus of the NTSC system under the illumination of a fluorescent lamp in an area where the power-supply frequency is 60 Hz.
In this case, whereas one field is 1/60 seconds, the period of the luminance change of the fluorescent lamp is 1/120 seconds. Consequently, during the normal shutter time, where the exposure time is 1/60 seconds, the amount of exposure becomes constant regardless of the exposure timing, and flicker, including in-plane flicker, does not occur. However, during a high-speed shutter time, where the exposure time is shorter than 1/60 seconds, as shown on the right side of FIG. 6A, flicker that completes in one field (one picture plane) and whose stripe pattern becomes the same in each field (each picture plane) occurs.
However, as shown in case 3 in FIG. 7, when the shutter is set to a high-speed shutter with an exposure time of 1/120 seconds, similarly to that during the normal shutter time with an exposure time of 1/60 seconds, the amount of exposure becomes constant regardless of the exposure timing, and flicker, including in-plane flicker, does not occur.
Case 4 of FIG. 6B shows a case in which a subject is photographed by a CMOS imaging apparatus of the PAL system under the illumination of a fluorescent lamp in an area where the power-supply frequency is 50 Hz.
In this case, whereas one field is 1/50 seconds, the period of the luminance change of the fluorescent lamp is 1/100 seconds. Consequently, during the normal shutter time, where the exposure time is 1/50 seconds, the amount of exposure becomes constant regardless of the exposure timing, and flicker, including in-plane flicker, does not occur. However, during a high-speed shutter time, where the exposure time is shorter than 1/50 seconds, as shown on the right side of FIG. 6B, flicker that completes in one field (one picture plane) and whose stripe pattern becomes the same in each field (each picture plane) occurs.
However, as shown in case 4 in FIG. 7, when the shutter is set to a high-speed shutter with an exposure time of 1/100 seconds, similarly to that during the normal shutter time with an exposure time of 1/50 seconds, the amount of exposure becomes constant regardless of the exposure timing, and flicker, including in-plane flicker, does not occur.
Then, when flicker, which is not continuous in the time axis, that completes in one field (one picture plane), occurs in case 3 or case 4 of FIG. 7, in which the shutter is set to a high-speed shutter, since a distinction between picture pattern components by the subject and fluorescent-lamp flicker components in the video signal from the imaging device cannot be made from the very beginning, it is not possible to reduce flicker components by the flicker reduction method using the above-described continuity of flicker.
FIG. 7 summarizes the foregoing. In FIG. 7, case 1, in which the vertical synchronization frequency is 60 Hz and the power-supply frequency is 50 Hz; case 2, in which the vertical synchronization frequency is 50 Hz and the power-supply frequency is 60 Hz; case 3, in which the vertical synchronization frequency is 60 Hz and the power-supply frequency is 60 Hz; and case 4, in which the vertical synchronization frequency is 50 Hz and the power-supply frequency is 50 Hz, are as described above.
FIG. 7 also shows a case in which the vertical synchronization frequency (the frame frequency in this case) is 30 Hz in the CMOS imaging apparatus of the progressive system.
As shown in case 5 in FIG. 7, when a subject is photographed under the illumination of a fluorescent lamp in an area in which the power-supply frequency is 50 Hz by using a CMOS imaging apparatus in which the vertical synchronization frequency is 30 Hz, whereas one vertical period is 1/30 seconds, the period of the luminance change of the fluorescent lamp is 1/100 seconds. Consequently, during the normal shutter time, where the exposure time is 1/30 seconds, and also during the high-speed shutter time, where the exposure time is shorter than 1/30 seconds, flicker having continuity in the time axis, that is, flicker whose repetition period is three vertical periods (three picture planes), occurs (when viewed continuously, it appears to flow in the vertical direction).
However, when the shutter is set to a high-speed shutter with an exposure time of 1/100 seconds, 1/50 seconds, or 3/100 seconds, which is an integral multiple of the period of the luminance change of the fluorescent lamp, the amount of exposure becomes constant regardless of the exposure timing, and flicker, including in-plane flicker, does not occur.
Furthermore, as shown in case 6 in FIG. 7, when a subject is photographed under the illumination of a fluorescent lamp in an area where the power-supply frequency is 60 Hz by using a CMOS imaging apparatus in which the vertical synchronization frequency is 30 Hz, whereas one vertical period is 1/30 seconds, the period of the luminance change of the fluorescent lamp is 1/120 seconds. Consequently, during the normal shutter time, where the exposure time is 1/30 seconds, the amount of exposure becomes constant regardless of the exposure timing, and flicker, including in-plane flicker, does not occur. However, during the high-speed shutter time, where the exposure time is shorter than 1/30 seconds, similarly to that during the high-speed shutter of case 3 and case 4, flicker that completes in one period (one frame in this case) and whose flicker stripe pattern becomes the same in each vertical period (each frame in this case) occurs.
However, when the shutter is set to a high-speed shutter with an exposure time of 1/120 seconds, 1/60 seconds, or 1/40 seconds, which is an integral multiple of the period of the luminance change of the fluorescent lamp, similarly to that during the normal shutter time with an exposure time of 1/30 seconds, the amount of exposure becomes constant regardless of the exposure timing, and flicker, including in-plane flicker, does not occur.
As described above, in case 3, case 4, or case 6, in which the shutter is set to a high-speed shutter, flicker that completes in one period (one picture plane) and whose stripe pattern becomes the same in each vertical period (each picture plane) occurs, excluding a case in which the shutter speed is set at a specific speed, and a distinction between picture pattern components and fluorescent lamp flicker components in the video signal from the imaging device cannot be made. As a result, it is not possible to reduce flicker components by the flicker reduction method using the above-described continuity of flicker.
For this reason, in these cases, the shutter speed (exposure time) of the high-speed shutter may be set to a speed at which flicker does not occur, that is, 1/120 seconds in case 4; 1/100 seconds in case 3; and 1/120 seconds, 1/60 seconds, or 1/40 seconds in case 6. Alternatively, rather than being set to a high-speed shutter, the shutter may be set to a normal shutter in which flicker does not occur. That is, in case 3, the shutter speed may be set to 1/60 seconds; in case 4, the shutter speed may be set to 1/50 seconds; and in case 6, the shutter speed may be set to 1/30 seconds.
However, in order to achieve the above, it is necessary to detect the power-supply frequency separately by some method.
For example, a method for detecting the power-supply frequency on the basis of the relationship between the period (wavelength) of a stripe pattern of the flicker and the vertical period (the reciprocal of the vertical synchronization frequency) of the imaging apparatus has been considered. However, when flicker that is not continuous in the time axis and that completes in one vertical period (one picture plane) occurs in the manner described above, since a distinction between picture pattern components and fluorescent lamp flicker components in the video signal from the imaging device cannot be made from the very beginning, it is not possible to detect the power-supply frequency.
Furthermore, there is a method for detecting the power-supply frequency by an external sensor. However, in this method, the size and the cost of the imaging apparatus system increase.
Furthermore, when the shutter is set to a normal shutter rather than being set to a high-speed shutter, there are problems in electronic camera-shake correction, as described below.
Most recent imaging apparatuses have a camera-shake correction function of an electronic image frame cutout type. In this camera-shake correction method, camera shake is detected by a camera shake sensor incorporated in the camera or by a motion vector which occurs in the image, and an area of an appropriate size at an appropriate position is cut out and output from the input image on the basis of the detected amount of camera shake, thereby correcting the image signal so that the output image is always seen as being stationary.
However, in such electronic camera-shake correction, camera shake between picture planes (between fields or between frames) can be corrected, but commonly called afterimage blur due to camera shake which occurs during an exposure period cannot be corrected from the viewpoint of principles. That is, although camera shake between picture planes can be reduced by camera-shake correction, since the afterimage blur remains as is, the image quality deteriorates by the unbalance thereof.
Then, in order to reduce this afterimage blur, it is recommended that the shutter be set to a high-speed shutter so as to decrease the exposure time. However, if the shutter speed is made too high, rough movement of the moving picture becomes conspicuous, and therefore, the shutter is set to a shutter speed with approximately 1/100 seconds at which the relationship of the above becomes just satisfactory.
As described above, the imaging apparatus having an electronic camera-shake correction function is set so that a high-speed shutter is realized automatically when camera-shake correction is ON. That is, in order to perform electronic camera-shake correction, a high-speed shutter is necessary, and-in order to solve the problem of the fluorescent-lamp flicker, the high-speed shutter cannot be omitted.